{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/search/image/image-retrieval-ebook/color-histograms/00-build-histograms.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/search/image/image-retrieval-ebook/color-histograms/00-build-histograms.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building Color Histograms\n",
    "\n",
    "To build our histograms we will be relying primarily on the OpenCV and Numpy libraries.\n",
    "\n",
    "* **OpenCV** is a popular **C**omputer **V**ision (CV) library, it's full of useful image load/save and manipulation functions.\n",
    "* **NumPy** is a very popular library focused on providing optimized numerical operations for multi-dimensional arrays.\n",
    "\n",
    "We `pip install` both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install opencv-python numpy datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get a set of images, we will use the `pinecone/image-set` dataset from HuggingFace datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "data = load_dataset('pinecone/image-set', split='train', revision='e7d39fc')\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inside the *image_bytes* feature we have base64 encoded bytes representation of the images, we can decode them into Numpy arrays using the Open-CV library like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from base64 import b64decode\n",
    "import cv2\n",
    "import numpy as np\n",
    "\n",
    "def process_fn(sample):\n",
    "    image_bytes = b64decode(sample['image_bytes'])\n",
    "    image = cv2.imdecode(np.frombuffer(image_bytes, np.uint8), cv2.IMREAD_COLOR)\n",
    "    return image\n",
    "\n",
    "images = [process_fn(sample) for sample in data]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can display images with matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.imshow(images[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "OpenCV loads images in a **B**lue **G**reen **R**ed (BGR) format, Matplotlib expects RGB, so we invert to get the true color image like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shape = images[0].shape\n",
    "shape, images[0][0, 0, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "rgb_image = np.flip(images[0], 2)\n",
    "\n",
    "shape = rgb_image.shape\n",
    "shape, rgb_image[0, 0, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that while the `shape` of the array has remained the same, the three values have been reversed (those three values are the BGR -> RGB values for a single pixel in the images).\n",
    "\n",
    "Now we can visualize in real color."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(rgb_image)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a Histogram the Slow Way\n",
    "\n",
    "We'll start by building a color histogram the slow way so we can understand what is actually happening.\n",
    "\n",
    "Let's take a look at image `0`, pixel `0`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "images[0][0, 0, :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each pixel has a **B**lue, **G**reen, and **R**ed activation value on a scale from `0` (no color) to `255` (max color).\n",
    "\n",
    "We can see this by plotting the following colors by simply modifying the RGB values (remember they are inverted from BGR to RGB between OpenCV and Matplotlib):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "blue = [0, 0, 255]\n",
    "green = [0, 255, 0]\n",
    "red = [255, 0, 0]\n",
    "violet = [255, 0, 255]\n",
    "orange = [255, 165, 0]\n",
    "yellow = [255, 255, 0]\n",
    "cyan = [0, 255, 255]\n",
    "\n",
    "colors = np.asarray([[blue, green, red, violet, orange, yellow, cyan]])\n",
    "\n",
    "plt.imshow(colors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From these numbers:\n",
    "\n",
    "| Blue | Green | Red |\n",
    "| --- | --- | --- |\n",
    "| 165 | 174 | 134 |\n",
    "\n",
    "We can estimate that the first pixel will be a slight blue-green color (as these both overpower the red value, albeit slightly). We can see the pixel here in the top-left corner:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(rgb_image[:3, :3, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can plot these RGB values for each image across all pixels, moving from left-right and top-bottom. But to do this we need to transform the image array into a vector (one vector per color channel, RGB)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "image_vector = rgb_image.reshape(1, -1, 3)\n",
    "image_vector.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can still see the top left three pixels are the same:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(image_vector[:, :3, :])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can split the values into their respective color channels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "red = image_vector[0, :, 0]\n",
    "green = image_vector[0, :, 1]\n",
    "blue = image_vector[0, :, 2]\n",
    "\n",
    "red.shape, green.shape, blue.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now visualize each with a histogram."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(1, 3, figsize=(15, 4), sharey=True)\n",
    "axs[0].hist(red, bins=256, color='r')\n",
    "axs[1].hist(green, bins=256, color='g')\n",
    "axs[2].hist(blue, bins=256, color='b')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see the three color channels, on the x-axis we have the pixel color value (from *0 -> 255*) and on the y-axis we see the total number of pixels with that color value.\n",
    "\n",
    "Let's put everything we've done so far into a single function so we can replicate these charts for a few images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_histogram(image, bins=256):\n",
    "    # convert from BGR to RGB\n",
    "    rgb_image = np.flip(image, 2)\n",
    "    # show the image\n",
    "    plt.imshow(rgb_image)\n",
    "    # convert to a vector\n",
    "    image_vector = rgb_image.reshape(1, -1, 3)\n",
    "    # break into given number of bins\n",
    "    div = 256 / bins\n",
    "    bins_vector = (image_vector / div).astype(int)\n",
    "    # get the red, green, and blue channels\n",
    "    red = bins_vector[0, :, 0]\n",
    "    green = bins_vector[0, :, 1]\n",
    "    blue = bins_vector[0, :, 2]\n",
    "    # build the histograms\n",
    "    fig, axs = plt.subplots(1, 3, figsize=(15, 4), sharey=True)\n",
    "    axs[0].hist(red, bins=bins, color='r')\n",
    "    axs[1].hist(green, bins=bins, color='g')\n",
    "    axs[2].hist(blue, bins=bins, color='b')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we will see a lot of blue in this image, and it shows\n",
    "# as high counts in high values in the blue histogram\n",
    "build_histogram(images[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing I added to this formula is the option to reduce the number of bins we are visualizing, eg rather than visualize every color value (0 -> 255) as a single bar, we can visualize values [0-4, 5-9, 10-14, ...] together using the `bins` parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "build_histogram(images[1], 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can be great for reducing the size of our image representations while still keeping as much information as possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# very little blue in this image, and it shows as high counts in *low* values\n",
    "# of the blue histogram\n",
    "build_histogram(images[2], 64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# blue again is very low, plenty of green\n",
    "build_histogram(images[3], 64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "build_histogram(images[4], 64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "build_histogram(images[5], 64)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Okay so that is the *slow way* of building histograms, hopefully it's very clear how this works. But how can we do this more efficiently?\n",
    "\n",
    "The open-cv library which we have already imported as `cv2` actually has a function specifically built for calculating histograms called `calcHist`, it looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "red_hist = cv2.calcHist([images[5]], [2], None, [64], [0, 256])\n",
    "green_hist = cv2.calcHist([images[5]], [1], None, [64], [0, 256])\n",
    "blue_hist = cv2.calcHist([images[5]], [0], None, [64], [0, 256])\n",
    "\n",
    "red_hist.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(cv2.calcHist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values used here are:\n",
    "\n",
    "```\n",
    "cv2.calcHist([images], [channels], [mask], [bins], [hist_range])\n",
    "```\n",
    "\n",
    "Where:\n",
    "* `images` is our cv2 loaded image with channels blue, green, red. This argument expects a list of images which is why we have placed our single image inside square brackets `[]`.\n",
    "* `channels` is the color channel (BGR) that we'd like to create a histogram for. We do this for a single channel at once.\n",
    "* `mask` is another image (or list of images) consisting of `0` and `1` values that allow us to *mask* part of the `images` if wanted, we don't want to do this so we set it to `None`.\n",
    "* `bins` is as before, the number of bins/ranges that we want to place our values in. We can set this to `256` if we'd like to keep the original values.\n",
    "* `hist_range` is the range of color values we would expect. As we're using RGB we expect a min value of `0` and max value of `255`, so we write `[0, 256]` (the upper limit is exclusive, not inclusive)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(1, 3, figsize=(15, 4), sharey=True)\n",
    "axs[0].plot(red_hist, color='r')\n",
    "axs[1].plot(green_hist, color='g')\n",
    "axs[2].plot(blue_hist, color='b')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13 ('image-embeds')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "9ec8fc8fb845fc3e050bf8bf651a355c069bbfeddee31167baf4bc42b6050476"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
